Linux System Programming

Linux System Programming Talking Directly to the Kernel and C Library.pdf

Table of ContentsForeword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvPreface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii1. Introduction and Essential Concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1System Programming 1Why Learn System Programming 2Cornerstones of System Programming 3System Calls 3The C Library 4The C Compiler 4APIs and ABIs 5APIs 5ABIs 6Standards 7POSIX and SUS History 7C Language Standards 8Linux and the Standards 8This Book and the Standards 9Concepts of Linux Programming 10Files and the Filesystem 10Processes 16Users and Groups 18Permissions 19Signals 20Interprocess Communication 20Headers 21Error Handling 21vGetting Started with System Programming 242. File I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Opening Files 26The open() System Call 26Owners of New Files 29Permissions of New Files 29The creat() Function 31Return Values and Error Codes 32Reading via read() 32Return Values 33Reading All the Bytes 34Nonblocking Reads 35Other Error Values 35Size Limits on read() 36Writing with write() 36Partial Writes 37Append Mode 38Nonblocking Writes 38Other Error Codes 38Size Limits on write() 39Behavior of write() 39Synchronized I/O 40fsync() and fdatasync() 41sync() 43The O_SYNC Flag 43O_DSYNC and O_RSYNC 44Direct I/O 45Closing Files 45Error Values 46Seeking with lseek() 46Seeking Past the End of a File 47Error Values 48Limitations 48Positional Reads and Writes 49Error Values 50Truncating Files 50Multiplexed I/O 51select() 52poll() 58poll() Versus select() 61Kernel Internals 62vi | Table of ContentsThe Virtual Filesystem 62The Page Cache 63Page Writeback 65Conclusion 663. Buffered I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67User-Buffered I/O 67Block Size 69Standard I/O 70File Pointers 70Opening Files 71Modes 71Opening a Stream via File Descriptor 72Closing Streams 73Closing All Streams 73Reading from a Stream 73Reading a Character at a Time 74Reading an Entire Line 75Reading Binary Data 76Writing to a Stream 77Writing a Single Character 78Writing a String of Characters 78Writing Binary Data 79Sample Program Using Buffered I/O 79Seeking a Stream 80Obtaining the Current Stream Position 82Flushing a Stream 82Errors and End-of-File 83Obtaining the Associated File Descriptor 84Controlling the Buffering 84Thread Safety 86Manual File Locking 87Unlocked Stream Operations 88Critiques of Standard I/O 89Conclusion 904. Advanced File I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Scatter/Gather I/O 92readv() and writev() 92Event Poll 97Creating a New Epoll Instance 97Controlling Epoll 98Table of Contents | viiWaiting for Events with Epoll 101Edge- Versus Level-Triggered Events 103Mapping Files into Memory 104mmap() 104munmap() 109Mapping Example 109Advantages of mmap() 111Disadvantages of mmap() 111Resizing a Mapping 112Changing the Protection of a Mapping 113Synchronizing a File with a Mapping 114Giving Advice on a Mapping 115Advice for Normal File I/O 118The posix_fadvise() System Call 118The readahead() System Call 120Advice Is Cheap 121Synchronized, Synchronous, and Asynchronous Operations 121Asynchronous I/O 123I/O Schedulers and I/O Performance 123Disk Addressing 124The Life of an I/O Scheduler 124Helping Out Reads 125Selecting and Configuring Your I/O Scheduler 129Optimzing I/O Performance 129Conclusion 1355. Process Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137Programs, Processes, and Threads 137The Process ID 138Process ID Allocation 138The Process Hierarchy 139pid_t 139Obtaining the Process ID and Parent Process ID 140Running a New Process 140The Exec Family of Calls 140The fork() System Call 145Terminating a Process 148Other Ways to Terminate 149atexit() 149on_exit() 151SIGCHLD 151Waiting for Terminated Child Processes 151viii | Table of ContentsWaiting for a Specific Process 154Even More Waiting Versatility 156BSD Wants to Play: wait3() and wait4() 158Launching and Waiting for a New Process 160Zombies 162Users and Groups 163Real, Effective, and Saved User and Group IDs 163Changing the Real or Saved User or Group ID 164Changing the Effective User or Group ID 165Changing the User and Group IDs, BSD Style 165Changing the User and Group IDs, HP-UX Style 166Preferred User/Group ID Manipulations 166Support for Saved User IDs 167Obtaining the User and Group IDs 167Sessions and Process Groups 167Session System Calls 169Process Group System Calls 170Obsolete Process Group Functions 172Daemons 172Conclusion 1756. Advanced Process Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177Process Scheduling 177Timeslices 178I/O- Versus Processor-Bound Processes 179Preemptive Scheduling 179The Completely Fair Scheduler 180Yielding the Processor 181Legitimate Uses 182Process Priorities 183nice() 183getpriority() and setpriority() 184I/O Priorities 186Processor Affinity 186sched_getaffinity() and sched_setaffinity() 187Real-Time Systems 190Hard Versus Soft Real-Time Systems 190Latency, Jitter, and Deadlines 191Linux’s Real-Time Support 192Linux Scheduling Policies and Priorities 192Setting Scheduling Parameters 196sched_rr_get_interval() 199Table of Contents | ixPrecautions with Real-Time Processes 201Determinism 201Resource Limits 204The Limits 205Setting and Retrieving Limits 2097. Threading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211Binaries, Processes, and Threads 211Multithreading 212Costs of Multithreading 214Alternatives to Multithreading 214Threading Models 215User-Level Threading 215Hybrid Threading 216Coroutines and Fibers 216Threading Patterns 217Thread-per-Connection 217Event-Driven Threading 218Concurrency, Parallelism, and Races 218Race Conditions 219Synchronization 222Mutexes 222Deadlocks 224Pthreads 226Linux Threading Implementations 226The Pthread API 227Linking Pthreads 227Creating Threads 228Thread IDs 229Terminating Threads 230Joining and Detaching Threads 233A Threading Example 234Pthread Mutexes 235Further Study 2398. File and Directory Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241Files and Their Metadata 241The Stat Family 241Permissions 246Ownership 248Extended Attributes 250Extended Attribute Operations 253x | Table of ContentsDirectories 259The Current Working Directory 260Creating Directories 265Removing Directories 267Reading a Directory’s Contents 268Links 271Hard Links 272Symbolic Links 273Unlinking 275Copying and Moving Files 277Copying 277Moving 278Device Nodes 280Special Device Nodes 280The Random Number Generator 281Out-of-Band Communication 281Monitoring File Events 283Initializing inotify 284Watches 285inotify Events 287Advanced Watch Options 290Removing an inotify Watch 291Obtaining the Size of the Event Queue 292Destroying an inotify Instance 2929. Memory Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293The Process Address Space 293Pages and Paging 293Memory Regions 295Allocating Dynamic Memory 296Allocating Arrays 298Resizing Allocations 299Freeing Dynamic Memory 301Alignment 303Managing the Data Segment 307Anonymous Memory Mappings 308Creating Anonymous Memory Mappings 309Mapping /dev/zero 311Advanced Memory Allocation 312Fine-Tuning with malloc_usable_size() and malloc_trim() 314Debugging Memory Allocations 315Obtaining Statistics 315Table of Contents | xiStack-Based Allocations 316Duplicating Strings on the Stack 318Variable-Length Arrays 319Choosing a Memory Allocation Mechanism 320Manipulating Memory 321Setting Bytes 321Comparing Bytes 322Moving Bytes 323Searching Bytes 324Frobnicating Bytes 325Locking Memory 325Locking Part of an Address Space 326Locking All of an Address Space 327Unlocking Memory 328Locking Limits 328Is a Page in Physical Memory? 328Opportunistic Allocation 329Overcommitting and OOM 33010. Signals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333Signal Concepts 334Signal Identifiers 334Signals Supported by Linux 335Basic Signal Management 340Waiting for a Signal, Any Signal 341Examples 342Execution and Inheritance 344Mapping Signal Numbers to Strings 345Sending a Signal 346Permissions 346Examples 347Sending a Signal to Yourself 347Sending a Signal to an Entire Process Group 347Reentrancy 348Guaranteed-Reentrant Functions 349Signal Sets 350More Signal Set Functions 351Blocking Signals 351Retrieving Pending Signals 352Waiting for a Set of Signals 353Advanced Signal Management 353The siginfo_t Structure 355xii | Table of ContentsThe Wonderful World of si_code 357Sending a Signal with a Payload 361Signal Payload Example 362A Flaw in Unix? 36211. Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363Time’s Data Structures 365The Original Representation 366And Now, Microsecond Precision 366Even Better: Nanosecond Precision 366Breaking Down Time 367A Type for Process Time 368POSIX Clocks 368Time Source Resolution 369Getting the Current Time of Day 370A Better Interface 371An Advanced Interface 372Getting the Process Time 372Setting the Current Time of Day 373Setting Time with Precision 374An Advanced Interface for Setting the Time 374Playing with Time 375Tuning the System Clock 377Sleeping and Waiting 380Sleeping with Microsecond Precision 381Sleeping with Nanosecond Resolution 382An Advanced Approach to Sleep 383A Portable Way to Sleep 385Overruns 385Alternatives to Sleeping 386Timers 386Simple Alarms 386Interval Timers 387Advanced Timers 389A. GCC Extensions to the C Language. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395B. Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411